docs(experiments): add model experiments spec by markijbema · Pull Request #3463 · Kilo-Org/cloud

markijbema · 2026-05-25T12:56:45Z

Summary

Add .specs/model-experiments.md as the durable source of truth for model experiment scope, routing, lifecycle, retention, reporting, and secret-handling rules.
Register the spec in AGENTS.md so future model-experiment changes read the business rules before editing the domain.

Verification

Not manually exercised; docs-only spec and registration change.

Visual Changes

N/A

Reviewer Notes

The spec is intentionally business-rule focused and excludes implementation-plan details unless they protect an invariant.

kilo-code-bot · 2026-05-25T12:58:02Z

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Docs-only PR adding the model-experiments spec; all four incremental commits are spec refinements with no logic or security concerns. The activation-minimum WARNING from the previous review is now resolved.

Incremental Changes Reviewed (all 4 commits)

docs(experiments): add model experiments spec — Initial spec file and AGENTS.md registration.
remove redundant info — Removed the "Status" (Draft) and "Conventions" (RFC 2119) boilerplate sections.
docs(experiments): loosen prompt capture cap requirement — Changed prompt-body cap from MUST to MAY, with conditional MUST rules applying only when truncation is used.
docs(experiments): allow single-variant activation — Changed activation minimum from "at least two variants" to "at least one variant". This resolves the previous WARNING that flagged a conflict with the implementation (which only rejects variants.length < 1). The PR author confirmed this is intentional: the spec should support sequential (single-variant) testing, not only A/B.

Previous Issues — Status

File	Issue	Status
`.specs/model-experiments.md` line 11	WARNING: Scope exclusions (BYOK, kilo-internal) not enforced by current routing/admin path	Open (acknowledged by author as intentional scoping)
`.specs/model-experiments.md`	WARNING: Activation minimum conflicts with current behavior (spec said ≥2, impl accepts ≥1)	Resolved — changed to "at least one variant" in latest commit

Files Reviewed (2 files)

.specs/model-experiments.md — 122-line spec; activation-minimum WARNING resolved
AGENTS.md — spec registration row; no issues

_{Reviewed by claude-sonnet-4.6 · 286,158 tokens}

_{Review guidance: REVIEW.md from base branch main}

jeanduplessis

Left two warning comments where the proposed source-of-truth spec appears stricter than current model-experiment behavior.

jeanduplessis · 2026-05-25T14:26:31Z

+
+Model experiments exist only to A/B test preview or otherwise experimental model checkpoints in partnership with model providers. They are not a general-purpose traffic-splitting or rollout mechanism for production models.
+
+An experimented `public_model_id` MUST be a dedicated preview or experiment id that users explicitly select. Production model ids MUST NOT be silently bucketed. Experimented ids MUST NOT be added to `kilo-auto` candidate sets, presets, or other automatic selection paths unless the spec is explicitly changed to allow that behavior.


WARNING: Scope exclusions are not enforced by the current routing/admin path.

Trace:

The spec adds hard guarantees that experimented IDs are dedicated preview/experiment IDs and that kilo-internal/... traffic is outside model-experiment routing.

Current admin/routing code only requires public_model_id to be a non-empty string; assertActivatable does not reject production IDs or kilo-internal/...; getProvider checks experiment membership before kilo-internal/... custom LLM routing.

An admin can create/activate an experiment for a production model ID or kilo-internal/... ID, after which membership causes the experiment path to run despite the new spec saying those states MUST NOT happen.

Impact: The merged spec would claim opt-in/preview-only and kilo-internal/... exclusion invariants that current code does not preserve. A misconfigured active experiment can silently reroute production or internal-model traffic and capture experiment attribution/prompts under a contract that says that traffic is out of scope.

This is true; i scoped this out. I think we might want to reconsider how exactly we serve these experiments (it might make sense to combine them with a fallback for when an experiment pauses for instance, and put them behind another model (like kilo-auto/free). However, we do these changes manually so i think we should always do this carefully.

jeanduplessis · 2026-05-25T14:26:31Z

+- `paused`: gateway returns a local model-unavailable response for the experimented public id and MUST NOT silently fall through to default model routing.
+- `completed`: historical and non-routing; removed from routing membership and eligible to coexist with a draft or active replacement for the same public id.
+
+Activation MUST validate that the experiment has at least two variants, every variant has positive weight, every variant has a current version effective at or before activation time, and no other active or paused experiment targets the same public id.


WARNING: Activation minimum conflicts with current behavior.

Trace:

The spec requires activation to validate at least two variants.

Current implementation rejects only variants.length < 1; the admin UI says “at least 1 variant”; and model-experiments-router.test.ts explicitly activates a one-variant experiment successfully.

The repository already permits and tests a one-variant active experiment, while the new source-of-truth spec says activation MUST reject it.

Impact: Future reviewers/implementers will rely on a false invariant. One-variant experiments can be activated today, so the experiment may not be an A/B test and cohort/reporting assumptions based on “at least two variants” can be wrong.

oh nice catch, this is actually wrong. Though we started out with a/b testing i also want this to be used with purely sequential testing

docs(experiments): add model experiments spec

70c8b46

markijbema added 2 commits May 25, 2026 16:00

remove redundant info

7b3a7ca

docs(experiments): loosen prompt capture cap requirement

c3400d0

markijbema enabled auto-merge (squash) May 25, 2026 14:10

markijbema disabled auto-merge May 25, 2026 14:15

markijbema mentioned this pull request May 25, 2026

docs(experiments): trim model experiment plan #3466

Merged

jeanduplessis approved these changes May 25, 2026

View reviewed changes

docs(experiments): allow single-variant activation

05780af

markijbema merged commit 2994039 into main May 25, 2026
14 checks passed

markijbema deleted the mark-model-experiments-spec branch May 25, 2026 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(experiments): add model experiments spec#3463

docs(experiments): add model experiments spec#3463
markijbema merged 4 commits into
mainfrom
mark-model-experiments-spec

markijbema commented May 25, 2026

Uh oh!

kilo-code-bot Bot commented May 25, 2026 •

edited

Loading

Uh oh!

jeanduplessis left a comment •

edited

Loading

Uh oh!

jeanduplessis May 25, 2026

Uh oh!

markijbema May 25, 2026

Uh oh!

jeanduplessis May 25, 2026

Uh oh!

markijbema May 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		Model experiments exist only to A/B test preview or otherwise experimental model checkpoints in partnership with model providers. They are not a general-purpose traffic-splitting or rollout mechanism for production models.

		An experimented `public_model_id` MUST be a dedicated preview or experiment id that users explicitly select. Production model ids MUST NOT be silently bucketed. Experimented ids MUST NOT be added to `kilo-auto` candidate sets, presets, or other automatic selection paths unless the spec is explicitly changed to allow that behavior.

Conversation

markijbema commented May 25, 2026

Summary

Verification

Visual Changes

Reviewer Notes

Uh oh!

kilo-code-bot Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Summary

Executive Summary

Uh oh!

jeanduplessis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeanduplessis May 25, 2026

Choose a reason for hiding this comment

Uh oh!

markijbema May 25, 2026

Choose a reason for hiding this comment

Uh oh!

jeanduplessis May 25, 2026

Choose a reason for hiding this comment

Uh oh!

markijbema May 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kilo-code-bot Bot commented May 25, 2026 •

edited

Loading

jeanduplessis left a comment •

edited

Loading